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Abstract 


From  its  inception  as  a  technology,  virtual  reality  has  promised  to  revolutionize  the  way 
we  interact  with  our  computers  and  each  other.  So  far,  the  reality  of  virtual  reality  has  not  lived 
up  to  the  hype.  This  paper  explores  what  the  state  of  virtual  reality  interface  technology  will  be 
in  the  future  by  analyzing  the  current  state  of  the  art,  forecasting  trends  in  areas  relevant  to 
virtual  reality  interface  research  and  development,  and  highlighting  the  barriers  to  providing 
virtual  reality  environments  that  are  immersive  and  interactively  indistinguishable  from  reality 
(strong  VR).  This  research  shows  that  the  evolutionary  pathway  of  virtual  reality  technology 
development  will  not  be  able  to  overcome  all  of  the  barriers  and  limitations  inherent  in  the 
current  generation  of  interfaces.  I  use  a  reverse  tree  methodology  to  explore  alternate  pathways 
to  achieve  strong  VR.  Brain-machine  interfaces  (invasive  and  non-invasive)  represent  the  most 
likely  pathway  that  will  lead  to  a  strong  VR  interface.  The  US  Air  Force  should  continue  to 
develop  common  VR  interface  technology  using  widely  available  interfaces,  but  should  increase 
its  funding  and  support  for  technologies  that  will  enable  enhanced  brain-machine  interfaces  to 
ensure  its  dominance  in  training  and  simulation  for  the  future. 
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Introduction 


Since  the  late  1980s,  virtual  reality  has  been  heralded  as  the  next  technology  that  will 
revolutionize  the  way  we  use  our  computers  in  the  future.  Many  experts  in  the  virtual  reality 
field  were  convinced  that  the  technology  would  rapidly  overcome  the  physical  barriers  it  faced. 
Their  enthusiasm  for  future  possibilities  created  expectations  from  fields  as  diverse  as  the 
entertainment  industry  and  the  United  States  government.  Many  of  the  potential  users  were 
persuaded  by  the  hype  to  invest  significant  resources  into  this  next  great  technology. 

But  the  return  on  their  investments  has  been  slow  to  be  realized.  Although  virtual  reality  has 
been  able  to  gain  a  foothold  in  specific  applications  and  industries,  it  has  not  been  widely 
accepted  as  a  means  for  interacting  with  your  computer.  The  problems  that  have  plagued  virtual 
reality  technology  vary  widely  from  extremely  limiting  and  cumbersome  interfaces  to  a  lack  of 
applications  that  are  able  to  take  advantage  of  the  medium.  The  early  hype  and  resulting  failure 
to  deliver  has  created  an  environment  where  the  mention  of  virtual  reality  future  elicits  responses 
that  are  extremely  skeptical  about  its  usefulness  for  anything  but  esoteric  applications. 

This  skepticism  is  not  without  merit,  but  in  order  to  explore  the  future  of  virtual  reality 
technology,  we  must  push  aside  skepticism  based  on  past  failures  and  extend  our  understanding 
of  the  fundamental  problems  with  the  current  technology  and  its  ability  to  deliver  in  the  future. 
We  must  be  able  to  cut  through  the  baseless  hype  surrounding  this  technology  and  discover 
where  the  real  issues  lie  and  analyze  them  to  realize  what  promise  virtual  reality  holds  for  the 
future.  This  will  allow  us  to  make  better  resource  decisions  that  will  enable  us  to  take  advantage 
of  the  gains  promised  by  virtual  reality,  while  remaining  grounded  by  its  limitations. 

This  area  is  important  to  the  US  Air  Force  specifically  because  of  its  application  to  so  many 
different  areas.  Virtual  reality  and  its  related  technologies  would  allow  better  training  at  lower 
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costs,  increased  situational  awareness  and  data  visualization  of  the  multi-sensor  battlespace,  and 
the  ability  for  mission  rehearsal  to  be  carried  out  using  more  realistic  environments.  Many  of 
these  applications  already  use  available  virtual  reality  interfaces,  but  in  order  to  maintain  our 
competitive  advantage,  we  must  continually  push  our  technological  envelope.  We  must  seek  out 
the  most  cost  effective  solutions  for  revolutionary  technological  change  to  ensure  our 
technological  advantage. 

In  the  area  of  virtual  reality,  the  primary  issue  is  that  in  order  to  create  the  ideal  virtual 
reality  system,  one  must  focus  on  the  interface  between  the  man  and  the  machine.  In  virtual 
reality,  the  interface  is  the  application.  It  is  the  interface  that  allows  the  man  to  become 
immersed  in  an  artificially  created  world.  It  is  the  interface  that  allows  him  to  interact  with  his 
artificial  environment.  It  is  the  interface  that  defines  the  limits  of  this  virtual  environment.  In 
order  to  improve  the  performance  and  usefulness  of  virtual  reality,  we  must  improve  the 
interface  between  the  human  and  the  machine. 

One  of  the  failures  of  virtual  reality  has  been  its  inability  to  create  a  generalized  interface 
that  will  be  useful  for  many  different  applications.  Most  of  the  research  into  improving  virtual 
reality  performance  has  centered  on  individual  pieces  of  the  interface.  The  research  has  been 
dedicated  to  creating  improvements  to  existing  interfaces  -  an  evolutionary  upgrade  in  the  best 
of  circumstances.  Although  this  incremental  strategy  has  improved  the  overall  usefulness  of  our 
current  generation  of  virtual  reality,  its  future  success  may  be  limited  by  oncoming  technological 
barriers.  Incremental  improvement,  even  when  coupled  with  exponential  increases  in  computing 
power  and  increasing  market  pressures,  will  not  be  able  to  shift  the  emphasis  from  virtual  reality 
to  virtual  reality. 
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In  order  to  overcome  these  barriers,  we  must  look  for  alternate  pathways  that  lead  to 
enhanced  human-machine  interfaces  able  to  create  more  natural  and  realistic  interactions.  These 
revolutionary  interfaces  should  be  able  to  overcome  the  limitations  of  the  current  generation  of 
virtual  reality  interfaces  and  allow  us  to  sustain  our  competitive  advantages  in  training  and 
mission  rehearsal.  In  addition,  these  revolutionary  advances  will  allow  for  new  and  emerging 
applications  of  our  computing  power  that  will  lead  to  gains  in  other  areas  that  will  allow  us  to 
continue  our  dominance  in  the  battlespace  of  the  future. 

Methodology 

In  the  next  section  of  the  paper  I  will  outline  a  useful  definition  for  virtual  reality.  I  will 
briefly  examine  the  concepts  that  are  contained  in  the  definition.  This  discussion  will  include  a 
spectrum  of  “virtualness”  that  will  provide  a  framework  to  analyze  current  and  future  interface 
technologies.  This  framework  can  be  used  to  classify  current  and  future  VR  systems  by  how 
well  they  replicate  immersive  and/or  interactive  environments. 

The  technological  analysis  will  begin  with  a  review  of  current  virtual  reality  (VR)  interfaces 
for  each  of  the  channels  that  correspond  to  the  human  senses.  This  section  will  also  outline  the 
primary  research  and  development  efforts  that  seek  to  improve  the  current  VR  interface.  This 
section  will  provide  a  basic  understanding  of  the  state  of  the  art  interface  for  today  and  near 
tomorrow. 

The  technological  forecast  will  be  achieved  through  a  combination  of  two  primary 
methodologies:  trend  analysis  and  a  reverse  tree.  My  primary  method  of  forecasting  will  be  a 
trend  analysis  of  computing  power,  market  and  industry  forces,  and  system  hardware  gains.  This 
analysis  should  provide  an  adequate  forecast  for  what  should  be  available  along  an  evolutionary 
pathway  using  today’s  current  technology.  In  order  to  capture  technologies  that  are  not  included 
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in  this  pathway,  I  will  use  an  offshoot  of  the  paths  and  trees  forecasting  methodology  to  identify 
alternate  pathways  that  may  achieve  revolutionary  gains  in  VR  interface  technology. 

The  last  section  of  this  paper  will  provide  conclusions  about  the  future  of  virtual  reality 
interfaces,  while  outlining  some  of  the  relevant  implications  for  the  fielding  of  this  technology.  I 
will  analyze  the  implications  of  having  these  advances  available  to  us  in  the  US  Air  Force. 
These  analyses  will  be  supplemented  by  three  appendices  that  provide  further  information  about 
related  applications,  military  uses  of  this  technology,  and  non-technological  issues  that  may 
influence  the  development  of  VR  interface  technology. 

What  is  Virtual  Reality? 

As  long  as  you  see  the  screen,  you  ’re  not  in  virtual  reality.  When  the  screen 
disappears,  and  you  see  an  imaginary  scene...  then  you  are  in  virtual  reality.  - 

Gabriel  D.  Ofeisch,  Emeritus  Professor  of  Educational  Technology  at 
Howard  University1 

The  word  virtual  has  been  overused  to  describe  just  about  anything  that  deals  with  a 
computer-virtual  banking,  virtual  MPF,  virtual  teleconference.  Adding  to  the  confusion  is  the 
overuse  of  the  term  virtual  reality  to  describe  any  number  of  different  computer  applications  that 
provide  an  artificial  visual  environment.  This  has  contributed  to  a  lack  of  understanding  of  what 
virtual  reality  is.  In  order  to  explore  the  future  of  virtual  reality  technology,  we  must  begin  with 
an  understanding  of  what  virtual  reality  really  is. 

The  DoD  definition  of  virtual  reality  is  “the  effect  created  by  generating  an  environment  that 
does  not  exist  in  the  real  world.  Usually,  a  stereoscopic  display  and  computer-generated  three- 
dimensional  environment  giving  the  immersion  effect.  The  environment  is  interactive,  allowing 
the  participant  to  look  and  navigate  about  the  environment,  enhancing  the  immersion  effect.”" 
Although  this  definition  has  elements  that  enhance  our  understanding  of  VR,  it  also  limits  our 
understanding  of  what  virtual  reality  is  by  specifying  hardware  that  is  used  to  create  the  “virtual” 
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effect.  Other  attempts  at  defining  virtual  reality  in  the  military  have  suffered  from  a  similar 
focus  on  current  hardware.  A  1996  SAASS  research  paper  defined  it  as  “the  combination  of 
real-time  3-D  computer  graphics  with  shading  and  texture  mapping,  high  resolution  stereoscopic 
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large  screen  or  head  mounted  displays,  along  with  novel  user  interfaces.” 

Both  of  these  definitions  limit  an  exploration  of  virtual  reality  by  focusing  too  narrowly  on 
the  hardware  aspects  of  the  current  generation  of  virtual  reality.  In  order  to  forecast  the  future  of 
virtual  reality,  one  must  start  with  a  more  generalized  definition.  At  its  core,  virtual  reality 
should  be  understood  to  be  “an  immersive,  interactive  experience  generated  by  a  computer.”4 
Virtual  reality  must  be  an  experience  that  allows  the  user  to  engage  with  his  environment 
through  natural  means  without  breaking  the  illusion  of  the  immersive  reality. 

Although  better,  this  definition  is  too  general  to  be  able  to  evaluate  the  technology  that  will 
or  could  be  available  to  create  an  immersive  environment.  The  most  helpful  definition  focuses 
on  virtual  reality  as  the  interface  between  the  computer-generated  environment  and  the  person 
who  is  experiencing  this  environment.  In  this  respect,  virtual  reality  may  be  seen  as  the  sixth- 
generation  of  human-computer  interface.5  The  interface  creates  an  alternate  reality  for  the  user 
that  allows  them  to  navigate  through  the  environment  and  interact  with  objects  as  they  would  in 
the  real  world.6  The  immersion  of  the  user  into  the  virtual  reality  is  carried  out  by  interfacing 
computer  generated  information  through  multiple  sensorial  channels  -  visual,  auditory,  haptic, 
etc.7  The  only  reason  we  know  anything  about  our  real  environment  is  due  to  the  unique  human 
senses  that  have  developed  as  our  interface  to  the  world.  Virtual  reality  interfaces  work  through 
these  human  interfaces  to  replace  reality  with  an  alternate  reality  generated  by  a  computer. 

When  we  refer  to  virtual  reality  throughout  the  remainder  of  this  paper,  it  will  refer  to  a 
computer  interface  that  involves  real-time  simulation  of  an  interactive,  immersive  environment 
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through  multiple  sensorial  channels.  This  definition  encompasses  all  of  the  relevant  facets  of  the 

other  definitions,  without  limiting  it  to  the  currently  available  hardware  or  interfaces. 

3 1’s  of  Virtual  Reality  and  the  Holy  Grail 

Have  you  ever  had  a  dream,  Neo,  that  you  were  so  sure  was  real?  What  if  you 
were  unable  to  wake  from  that  dream?  How  would  you  know  the  difference 
between  the  dream  world  and  the  real  world?  -  Morpheus  (The  Matrix)8 

Using  the  definition  derived  in  the  previous  section  there  are  three  characteristics  that  allow 
us  to  evaluate  the  perfonnance  of  any  virtual  reality  system.  One  author  refers  to  these 
characteristics  as  the  3  I’s  of  virtual  reality.  These  characteristics  are  immersion,  interaction,  and 
information  intensity.9 

Immersion  is  the  ability  for  the  user’s  senses  to  be  isolated  from  the  real  world  and  for  the 
computer  generated  environment  to  appear  naturally  enough  for  the  user  to  be  “transported”  to 
another  place.10  Immersion  allows  the  user  to  forget  that  they  are  experiencing  an  artificial 
reality.  The  degree  of  immersion  of  any  VR  system  will  be  dependent  not  only  on  the  type  of 
interface  that  is  used,  but  will  also  be  a  result  of  the  number  and  type  of  user  senses  that  are 
being  stimulated.1 1  Immersion  then  is  a  measure  of  how  well  the  computer  can  replicate  sensory 
information  for  a  single  channel  as  well  as  the  number  of  sensory  channels  involved. 

Interactivity  is  the  ability  for  the  user  to  change  the  computer  generated  environment  in  real 
time.  Interactivity  is  what  allows  the  user  to  affect  the  scene  that  they  are  being  provided  from 
the  computer.  The  degree  of  interactivity  is  a  measure  of  not  only  the  speed  that  the  computer 
reacts  to  the  user’s  input,  but  also  the  ability  for  the  computer  to  respond  to  a  user’s  natural  form 
of  input  or  output.  More  interactive  VR  environments  will  not  only  allow  the  user  to  change  the 
visual  scene  by  moving  their  head  or  body,  but  will  also  provide  tactile  feedback  and  control  of 
objects  in  the  environment  through  their  hands.  The  most  interactive  VR  environments  would 
allow  the  user  to  handle  any  object  in  a  natural  manner. 
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Although  the  words  information  intensive  are  not  used  in  our  definition  of  virtual  reality,  the 
concept  they  represent  is  present.  Virtual  reality  is  information  intensive  to  support  its  ability  to 
be  immersive  and  interactive.  The  computer  must  be  able  to  react  to  the  inputs  of  the  user  and 
update  the  environment  appropriately  with  a  level  of  infonnation  that  does  not  break  the  illusion 
of  reality.  The  information  generated  by  the  computer  must  conform  to  all  of  the  sensorial 
channels  of  the  user  -  which  creates  an  extremely  intensive  data  processing  requirement  for  any 
input/output  channels. 

In  order  to  compare  and  analyze  the  current  VR  interfaces  as  well  as  any  future  advances, 
we  must  provide  a  framework  for  comparison.  The  degree  that  the  virtual  reality  interface 
conforms  to  the  characteristics  of  immersion  and  interactivity  can  be  plotted  in  Figure  1  below. 
Strong  VR  is  a  VR  interface  that  is  highly  immersive  and  highly  interactive  -  making  it  difficult 
to  distinguish  from  reality.  Weak  VR  corresponds  to  a  computing  environment  that  is  neither 
immersive  nor  interactive  in  a  natural  sense. 


Interactive 


Non-lmmersive 


Weak  VR 


Strong  VR 


Non-Interactive 


Figure  1 

The  so  called  “holy  grail”  of  virtual  reality  interfaces  is  to  have  the  user  so  immersed  in  an 
interactive  environment  that  they  are  unable  to  tell  the  difference  between  reality  and 
simulation.  This  type  of  virtual  reality  has  only  yet  been  achievable  in  the  imaginations  of 
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movie  writers  as  seen  in  The  Matrix  and  The  Thirteenth  Floor.  Many  of  the  early  adopters  of 
VR  technology  were  eager  to  say  that  this  type  of  VR  was  “not  too  far  off.”  Yet  most  of  the 
current  technologies  for  VR  interface  can  be  found  either  in  the  immersive/non-interactive  or 
non-immersive/interactive  quadrants.  The  challenge  for  our  future  is  to  discover  what  interface 
technology  will  be  technologically  feasible  and  that  we  should  pursue  in  order  to  allow  us  to 
achieve  the  “holy  grail.” 

Channels  of  Virtual  Reality  Interface 

In  virtual  reality,  the  smallest  details  of  sensory  experience,  the  building  blocks  of 
human  reality,  are  used  to  create  environments. — Ken  Pimentel  and  Kevin 
Teixeira14 

According  to  our  definition  of  virtual  reality,  interfaces  must  provide  immersion  and 
interactivity  on  multiple  sensory  channels.  Most  of  the  research  and  development  has 
historically  focused  on  the  two  channels  of  sight  and  sound.  Recently  VR  developers  have 
begun  to  concentrate  on  the  sense  of  touch  through  haptic  interface  technology.  Almost  no  VR 
systems  include  interfaces  for  the  senses  of  smell  or  taste.  In  order  to  decide  the  best  pathways 
to  approach  the  “holy  grail”  of  strong  VR,  we  must  evaluate  the  current  state  of  the  art  for  each 
of  these  sensory  channels  individually,  and  then  evaluate  their  use  in  VR  systems  that  attempt  to 
combine  multiple  sensory  channels. 

Visual 

The  visual  channel  in  virtual  reality  interfaces  should  provide  imagery  to  the  human  eye  that 
is  “indistinguishable  from  that  encountered  in  everyday  experience  in  terms  of  quality,  update 
rate,  and  extent.”15  In  order  to  provide  an  experience  that  is  both  immersive  and  interactive,  VR 
developers  have  used  a  number  of  different  interfaces  to  fulfill  the  visual  requirements.  Their 
early  efforts  evolved  from  the  use  of  CRTs  that  provided  little  immersive  reality  and  interaction 
was  limited  to  keyboard  or  mouse  inputs.  Although  the  current  technologies  provide  a  much 
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greater  immersive  and  interactive  experience,  there  is  still  a  large  gap  between  visual  displays 
and  visual  reality. 

The  main  effort  for  providing  visual  immersion  into  virtual  reality  has  been  through  head- 
mounted  devices  (HMDs).16  The  first  attempt  to  create  an  HMD  used  two  large  CRTs  in  a 
device  that  was  too  heavy  for  the  human  head  to  bear  without  external  support.17  Most  current 
HMDs  still  use  two  visual  displays  placed  directly  in  front  of  the  user’s  eyes  that  are  held  in 
place  by  the  device  which  simultaneously  tracks  the  movement  of  the  user’s  head  to  provide  cues 
to  change  the  visual  display.  Development  of  HMD  technology  has  been  aided  by  the  decreasing 
size  of  visual  display  technology  and  the  increasing  resolutions  available  on  these  same  displays. 
Current  generations  of  HMDs  are  being  designed  to  operate  within  the  form  factor  of  a  pair  of 
eyeglasses.18 

These  systems  are  limited  in  their  ability  to  provide  an  immersive  environment  by  their  field 
of  view  (FOV)  and  their  resolution  in  relation  to  visual  acuity.  Generally  speaking,  the  larger  the 
FOV  the  HMD  provides,  the  greater  the  immersive  experience.  But  increased  FOV  usually 
requires  a  tradeoff  in  resolution  which  can  degrade  the  immersive  quality  of  the  display.  In 
addition,  most  HMDs  on  the  market  provide  the  same  image  to  the  visual  display  in  each  eye.19 
This  decreases  the  complexity  of  the  device,  at  the  expense  of  stereoscopic  vision.  These 
tradeoffs  lead  most  developers  to  design  HMDs  that  are  application  specific  in  order  to  focus  on 
the  visual  needs  of  their  particular  virtual  interface. 

An  offshoot  of  the  current  HMD  technology  replaces  the  visual  displays  in  front  of  the 
user’s  eyes  with  a  device  that  is  able  to  write  the  visual  image  directly  on  the  user’s  retina. 
These  are  known  as  virtual  retinal  displays  or  retinal  scanning  displays.20  They  are  best  used  in 
augmented  reality  interfaces,  since  they  do  not  immerse  the  user  into  a  space  separate  from 
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reality.  The  user  is  able  to  see  the  real  scene  around  them,  but  with  the  ability  for  the  computer 
to  generate  information  to  be  overlaid  on  the  scene. 

Although  HMDs  are  the  primary  interface  of  choice  for  individual  immersive  applications, 
others  have  been  developed  that  provide  immersion  for  groups  of  users  at  one  time.  The  CAVE 
technology,  developed  by  the  University  of  Illinois  in  the  early  1990s,  uses  a  large  room  with 
graphics  being  projected  onto  two  or  three  walls  and  the  floor.  ~  The  users  must  still  wear  head 
mounted  devices  to  allow  them  to  see  the  displays  in  stereoscopic  vision,  but  this  technology 
frees  them  to  move  throughout  the  virtual  reality  scene  unencumbered  by  tethers  to  a  computer 
generated  display.  It  also  allows  multiple  users  to  survey  the  same  scene  simultaneously  without 
generating  multiple  visual  channels  of  information. 

Another  recent  technology  that  has  gathered  some  interest  for  use  in  the  visual  channel  of 
VR  interfaces  is  holography.  This  is  another  way  to  create  an  immersive  environment  that  is 
shared  by  multiple  users  simultaneously."-  This  technology  is  opposite  of  the  early  VR  thinking 
of  providing  resolution  in  a  limited  FOV.  Holography  attempts  to  produce  a  3-D  image  with 
resolution  that  is  not  limited  to  one  sight  line.  Although  it  took  the  first  inventor  20  years  to 
create  a  white  light  hologram,"  the  amount  of  research  being  applied  to  this  particular  area  has 
rapidly  increased  its  capabilities  for  use  in  VR  interfaces  in  recent  years. 

Even  though  the  technology  for  VR  interface  via  this  channel  has  received  the  primary 
weight  of  effort  from  VR  research,  it  still  has  significant  limitations  that  hamper  its  ability  to 
provide  an  immersive,  interactive  environment.  Most  of  these  systems  are  not  designed  to  take 
account  of  the  dual  nature  of  the  human  eye  for  tracking  and  focusing  on  objects  individually, 
and  few  allow  for  true  stereoscopic  vision  cues.  The  tradeoffs  required  between  FOV  and 
resolution  limits  the  immersive  nature  of  the  interface  due  to  the  physical  nature  of  the  display 
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devices.  Immersion  can  also  be  disrupted  through  any  perceived  lag  in  visual  feedback  greater 
than  250  ms.24  Most  of  the  visual  interfaces  are  unable  to  provide  a  seamless  immersive 
experience  indistinguishable  from  reality  without  lag  due  to  lack  of  processing  power  in  the 
computer  video  generation.  Finally,  all  of  these  visual  technologies  may  do  an  adequate  job  of 
providing  a  visually  immersive  experience,  but  none  of  them  is  conducive  to  providing  a 
simultaneously  interactive  environment  without  an  encumbering  haptic  interface  device. 

Auditory 

The  auditory  channel  has  not  received  as  much  emphasis  as  the  visual  channel,  but  does  not 
present  the  same  level  of  challenges  that  the  visual  channel  contained.  The  auditory  channel 
attempts  to  create  an  aurally  immersive  environment  that  is  also  interactive  in  that  the  perception 
of  the  sound  changes  as  the  user  changes  their  position  relative  to  the  source  of  the  sound.  There 
are  two  primary  mechanisms  for  introducing  the  aural  channel  to  the  VR  user:  headphones  or 
speakers. 

Headphones  are  the  preferred  method  for  providing  the  aural  information  to  the  user.  ~  This 
interface  allows  the  computer  to  generate  two  different  acoustic  waveforms  that  can  be  made  to 
correspond  to  the  ear  position  of  the  user  in  the  virtual  environment.  The  differences  in  the 
waveforms  are  used  to  provide  azimuth  perception,"  and  the  attenuation  of  the  waveform  is  used 
to  provide  distance  perception.  Currently,  the  technology  is  able  to  provide  aural  cues  that  allow 
the  user  to  resolve  information  received  via  headphones  with  an  accuracy  that  closely 
approximates  reality.  The  drawback  to  headphones  is  the  need  to  wear  a  device  on  your  head  or 
in  your  ears  that  detracts  from  the  immersive  experience. 

Speakers  allow  the  user  to  be  free  from  the  tether  of  an  interface  device  connected  to  their 
person,  but  are  unable  to  provide  the  same  levels  of  aural  resolution  as  headphones.  Particularly, 
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speakers  are  unable  to  simulate  a  point  source  of  sound  that  is  remotely  located  from  the 
speakers’  locations.  This  degrades  the  ability  for  the  user  to  explore  their  aural  environment 
through  the  positioning  of  their  head. 

Auditory  information  is  able  to  enhance  the  experience  of  the  user  by  interacting  with 
other  channels,  specifically  the  visual  interface.  Sounds  help  the  user  to  understand  the  visual 
scene  that  is  presented  to  them.  It  can  also  enhance  their  perception  of  the  quality  of  the  visual 
image  by  improving  the  quality  of  the  sound.  An  MIT  study  used  two  side-by-side  TV  sets 
showing  identical  images  with  differing  qualities  of  sound.  People  consistently  rated  the  TV  set 
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with  the  better  sound  as  having  a  much  better  picture. - 
Haptic 

The  haptic  channel  is  the  least  developed  of  the  primary  interface  mechanisms.  This  may  be 
due  to  the  inherent  complexity  of  “combining  sensory  functions  with  manipulative  functions  and 
with  the  use  of  electromechnical  systems.”-  Compared  to  the  previous  two  channels,  haptic 
technologies  must  be  able  to  provide  a  wide  variety  of  feedback  and  input  mechanisms.  Visual 
and  auditory  sensations  are  each  filtered  through  human  organs  that  are  specifically  designed  for 
their  function.  Sensations  of  force  and  movement  are  inseparable  from  actual  physical  contact 
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because  the  sensations  can  occur  at  any  part  of  the  human  body  -  both  internal  and  extemal.- 

Unlike  the  visual  and  auditory  channels,  the  haptic  channel  is  also  able  to  both  sense  what  is 

happening  around  the  user  and  act  on  that  environment  at  the  same  time.  This  ability  is  related 

to  the  way  the  human  body  processes  haptic  information  provided  by  its  environment. 

Haptic  sense  is  generally  divided  into  two  distinct  sensory  modalities:  the 
kinaesthetic  sense  which  includes  perception  of  muscular  effort  as  well  as 
proprioception,  and  the  tactile  sense  which  provides  cutaneous  infonnation  related 
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to  contact  between  the  skin  and  the  external  environment. 
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In  addition  to  the  sensation  of  touch  with  an  external  object,  this  sense  also  includes  the 
complexities  of  temperature,  stickiness,  texture,  and  frequency.  These  sensory  inputs  do  not 
even  take  into  account  the  body’s  ability  to  receive  haptic  information  through  its  vestibular 
system.  “At  the  same  time,  any  haptic  interface  must  not  only  provide  a  realistic  output  of  what 
the  body  should  be  feeling,  but  it  must  accept  the  inputs  from  the  human  interface  to  provide 
updates  to  the  environmental  situation  in  virtual  space. 

The  number  and  type  of  solutions  for  virtual  reality  interface  devices  continues  to  increase 
as  each  developer  creates  a  unique  solution  for  their  particular  application.  The  majority  of 
haptic  interfaces  in  use  today  are  force-feedback  joysticks  and  pads  used  in  computer  games  and 
motion  simulators.  The  primary  interface  device  for  VR  systems  currently  in  use  is  the  data 
glove  that  is  used  as  an  input  device  by  measuring  not  only  the  location  of  the  user’s  hand,  but 
also  the  degree  of  bend  or  flex  in  each  finger  joint.34  This  data  glove  may  be  married  with  an 
exoskeleton  to  provide  an  output  capability  as  well.  The  limited  uses  for  these  widely  fielded 
haptic  interfaces  highlights  the  difficulties  in  replicating  haptic  interactions  using  existing 
technology. 

Other  technologies  that  have  been  developed  to  allow  an  expansion  of  the  haptic  interface 
beyond  these  limited  uses  are  all  very  specialized  in  nature.  Some  of  these  technologies  are 
dedicated  to  creating  the  illusion  of  force  generated  by  a  virtual  object  like  exoskeletons,  tool¬ 
handling  force  displays  utilizing  specific  objects  for  specific  application  interface,  and  object- 
oriented  force  displays  that  are  able  to  move  or  deform  to  simulate  different  shapes.  Other 
technologies  are  used  to  simulate  the  full-body  haptics  involved  with  movement  or  locomotion 
within  the  virtual  environment.  These  devices  are  usually  treadmills,  foot  pads,  or  pedaling 
devices  that  allow  the  user  to  more  naturally  move  throughout  the  virtual  landscape.  Finally, 
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some  of  these  specialized  technologies  focus  on  the  sensation  an  object  creates  on  the  user’s 
skin.  A  micropin  array  has  been  created  that  is  able  to  convey  complex  textures  in  a  two- 
dimensional  geometry.36 

All  of  these  solutions  are  quite  capable  of  providing  reasonable  haptic  interfaces  for 
specialized  applications,  but  none  of  them  are  able  to  provide  a  full-body  immersive  experience 
that  accurately  replicates  the  human  experience  in  reality.  These  haptic  interfaces  are  limited  not 
only  by  the  size  and  weight  of  the  actuators  needed  to  provide  the  illusion  of  force,  but  are  also 
limited  by  the  VR  systems  “ability  to  coordinate  and  calculate  haptic  feedback  in  real-time  that 
match  the  human  perceptual  capabilities  in  accuracy  and  resolution.”  The  difficulty  in 
overcoming  these  barriers,  the  complexity  of  the  entire  haptic  system  of  the  human  body,  and  the 
lack  of  basic  research  in  haptic  sensations  makes  this  channel  the  least  likely  to  see  significant 
progress  toward  the  idea  of  “holy  grail”  virtual  reality. 

Taste  /  Smell 

The  channels  of  taste  and  smell  have  not  historically  been  included  in  any  of  the 
commercially  developed  VR  systems.  This  exclusion  has  generally  been  due  to  their  inability  to 
deliver  content  that  is  necessary  for  the  most  basic  virtual  reality  simulations.  These  senses 
provide  information  to  the  human  body  during  specific  applications,  but  are  not  generally  used  to 
orient  the  person  in  their  environment.  They  can,  however,  be  used  to  enhance  the  immersive 
quality  of  an  environment.  Walt  Disney  World  uses  the  sense  of  smell  during  their  virtual  reality 
ride  “Soaring”  to  further  immerse  the  rider  into  their  illusion. 

Both  of  these  channels  suffer  from  a  general  lack  of  knowledge  about  the  human  system 
underlying  the  sense  as  well  as  how  to  replicate  these  sensations  among  different  people. 
Additionally,  the  human  olfactory  system  is  often  strictly  tangled  with  other  sensory  channels. 
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One  author  cites  that  humans  are  able  to  identify  barely  one  third  of  odors  if  lacking  inputs  from 
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the  other  senses. 

Both  of  these  channels  are  difficult  to  deliver  to  the  user  with  any  accuracy  of  resolution. 
With  the  olfactory  sense  it  is  difficult  to  regulate  the  dispersion  of  the  scent  and  the  evacuation  of 
the  scent  when  no  longer  required.  ’9  Similarly,  the  sense  of  taste  is  difficult  to  introduce  to  the 
user  without  having  a  direct  interface  with  their  tongue.  This  would  require  an  intrusive 
interface  device  that  would  degrade  the  overall  immersive  experience  required  for  strong  VR. 

VR  Interface  Systems 

Figure  2  summarizes  where  each  of  the  current  technologies  lies  within  our  framework  for 
VR  interface  comparisons. 
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Figure  2  Current  Technologies  on  VR  Framework 

As  one  can  see  from  the  discussion  of  the  individual  channels,  each  is  able  to  contribute  its 

own  strengths  to  the  entire  VR  experience.  .  But  it  is  only  through  the  synergy  of  multi-channel 

sensory  experience  that  strong  VR  may  be  approached. 

“Virtual  reality  is  more  than  the  sum  of  its  components;  it  is  fundamentally  a 
system  technology.  The  components  have  no  significance  except  as  they 
contribute  to  a  functioning  whole.”40 
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The  search  for  convergence  and  synergy  will  continue  to  drive  the  VR  interfaces  described  along 
an  evolutionary  pathway  that  will  lead  VR  closer  to  the  “holy  grail”.  But  these  gains  will  also 
require  that  certain  tradeoffs  continue  to  be  made  as  they  have  been  made  throughout  virtual 
reality’s  short  history.  In  order  to  understand  how  much  these  convergent  interfaces  will  be  able 
overcome  the  current  limitations,  we  must  understand  what  technological  trends  we  expect  to 
find  that  may  influence  the  development  of  more  robust  VR  interface  solutions. 

Technological  Trends 
Computing  Power 

One  of  the  primary  drivers  for  advances  in  the  computing  industry  for  most  of  its  history  has 
come  to  be  known  as  Moore’s  law.  This  “law”  states  that  computing  power  (measured  by 
number  of  transistors  on  a  chip)  doubles  every  24  months.  This  has  held  true  from  its  first 
statement  in  the  mid-1960s  through  today  as  shown  in  Figure  2.  This  constant  doubling  of 
computer  processing  power  and  speed  is  also  accompanied  by  a  reduction  in  the  cost  and  size  of 
comparable  units.  Both  of  these  principles  expose  the  exponential  growth  of  computing  power 
and  will  have  significant  impacts  on  the  future  of  any  computer-based  technology  in  2025  if 
these  continue  to  hold  true. 
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Figure  3  Moore’s  Law41 

Many  of  the  indicators  for  the  growth  of  computing  power  continue  to  support  Moore’s  law 
through  at  least  2025.  A  RAND  study  in  2006  estimated  that  Moore’s  law  should  remain  in 
effect  through  2015  and  most  likely  would  continue  through  2020.  Although  some  experts 
have  estimated  that  it  will  reach  a  physical  limit  based  on  chip  construction  technology,  current 
research  efforts  in  nanotechnology  and  quantum  computing  should  allow  Moore’s  law  to 
continue  beyond  the  current  generation  of  chips. 

As  this  continues  to  hold  true,  computers  in  the  next  15  years  should  continue  to  become 
smaller,  faster  and  cheaper  to  employ.  Their  storage  space  should  also  experience  exponential 
growth  within  the  same  RAM  form  factors.  Both  of  these  factors  will  continue  to  drive 
computers  to  become  more  portable  as  the  processing  power  can  be  pushed  from  a  central 
processing  unit  to  individual  components.  Portable  devices  will  be  able  to  contain  the  computing 
power  of  current  chip  technologies  and  storage  mediums. 

How  will  this  continued  growth  of  computing  power  affect  the  virtual  reality  interfaces? 
One  author  provides  a  critical  link  between  computing  power  and  human-machine  interfaces. 
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“The  evolution  of  computing  has  been  so  fast  that  we’ve  only  recently  had  enough  low-cost 
computing  power  to  spend  it  freely  on  improving  the  ease  of  interaction  between  you  and  your 
computer.”43  As  computers  become  faster,  there  is  more  overhead  in  the  system  to  build  a  better 
interface  that  is  more  natural  and  intuitive.  This  has  been  true  in  the  past  with  the  change  of 
computer  interfaces  growing  along  with  increased  computing  power  from  punch  cards  to 
command  line  to  GUI. 

Although  better  processing  speed  has  enabled  more  intuitive  interfaces,  it  has  not  had  the 
same  effect  on  the  general  virtual  reality  systems  over  the  past  15  years.  As  shown  in  Figure  2, 
Moore’s  law  was  in  effect  over  this  period  of  time,  but  the  advances  in  virtual  reality 
technologies  have  not  grown  at  a  corresponding  rate.  In  fact,  most  of  the  virtual  reality 
interfaces  in  use  today  are  merely  evolutions  of  interfaces  that  have  been  in  production  since  the 
start  of  the  industry.  Moore’s  law  does  not  appear  to  have  a  one-to-one  effect  on  the  growth  or 
development  of  virtual  reality  interfaces,  but  it  has  allowed  for  an  increased  level  of  immersive 
experience  by  dramatically  changing  the  graphics  capabilities  available  in  general  production 
computers.  From  these  examples  one  must  conclude  that  Moore’s  law  will  have  some  ability  to 
drive  virtual  reality  technology  development,  but  may  not  be  the  primary  driver  for  enhanced 
interface  technology. 

Market  /  Industry 

Another  factor  that  may  drive  innovations  in  virtual  reality  interfaces  over  the  next  1 5  years 
is  the  market  demand  for  these  devices.  Figure  3  shows  the  past  growth  and  a  forecast  of 
industry  growth  through  2008.  The  overall  trend  is  for  15-20%  growth  per  year.  Industry 
analysts  expect  the  growth  in  particular  areas  of  the  virtual  reality  industry  to  experience 
significantly  higher  growth  in  research  and  production.  Using  these  numbers  we  can  begin  to  get 
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an  idea  of  the  various  stages  of  development  that  areas  of  the  VR  interface  industry  may  be 
currently  in.  These  will  help  us  to  forecast  the  trend  toward  future  spending  and  development  in 
each  area. 


Figure  4  Growth  and  Forecast  of  VR  market  $ 

In  his  work  on  technological  forecasting,  Joseph  Martino  presents  a  model  to  explain  the 
progression  of  technology  from  research  to  production.  His  model  suggests  that  for  any 
technology  one  can  identify  its  different  stages  by  the  number  of  documents  produced 
concerning  that  particular  stage.44  I  suggest  that  these  same  stages  may  be  discovered  by 
examining  the  dollar  amounts  spent  on  each  area.  His  model  defines  the  stages  as:  basic 
research,  applied  research,  development,  application,  and  social  impact.  Using  the  market  data 
for  the  VR  interface  industry,  it  appears  that  the  majority  of  the  industry  is  still  somewhere 
between  the  applied  research  and  development  phases,  although  there  are  no  certain  indicators 
and  the  technologies  themselves  may  go  back  and  forth  between  phases. 

The  market  value  in  Figure  3  includes  the  overall  spending  on  VR  interfaces  as  a  whole, 
including  research  and  development  of  future  technologies.  The  values  for  revenue  are 
specifically  those  dollars  brought  in  by  the  industry  from  applications  and  fully  functional 
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interface  devices.  From  the  growth  shown  (and  forecast)  on  the  chart,  the  spending  on  research 
and  development  should  continue  to  outpace  the  creation  of  applied  interfaces  and  the  revenues 
they  bring  in.  The  area  of  greatest  interest  for  the  industry  for  research  and  development  is  the 
area  of  usable  haptic  interfaces.45 

These  trends  tend  to  suggest  that  we  have  yet  to  see  the  greatest  improvements  in  VR 
interface  technology.  The  efforts  of  the  basic  and  applied  research  will  not  see  development  and 
application  for  5-10  years.  The  levels  of  spending  suggest  that  the  industry  as  a  whole  will 
continue  to  value  incremental  improvements  in  current  VR  interface  technologies,  while 
continuing  to  search  for  new  and  better  pathways  for  future  applications. 

The  one  area  of  the  market  that  has  seen  the  most  rapid  growth  in  application  and  societal 
impact  is  that  of  simple  haptic  interface  devices  like  force  feedback  joysticks  and  other  gaming 
devices.  Although  there  have  been  attempts  at  introducing  VR  interfaces  into  mainstream  media 
since  the  early  1990s,  none  have  seen  the  rapid  adoption  by  the  public  like  the  new  Nintendo 
Wii.  This  gaming  system  is  the  most  mature,  user-friendly  gaming  interface  system  and  has 
received  widespread  attention  for  its  intuitive  interface  and  engrossing  game  play.  This  adoption 
of  a  mature  VR  interface  technology  bodes  well  for  the  future  research  and  development  of 
improved  systems  of  this  type. 

All  of  these  industry  and  market  trends  suggest  that  entertainment  dollars  will  continue  to 
drive  the  types  of  VR  interfaces  as  well  as  their  adoption  rates.  The  industry  will  continue  to 
grow  and  adapt  to  new  market  spaces  while  finding  new  applications  for  existing  technologies 
already  in  production.  The  economic  potential  of  VR  interface  devices  will  continue  to  attract 
investment  by  new  companies  with  new  ideas  for  creating  user-friendly  immersive,  interactive 
environments. 
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The  market  and  industry  drivers  for  VR  interfaces  should  follow  a  familiar  path  of 
development.  Earlier  technologies  have  begun  as  specialized  systems  that  were  prohibitively 
expensive  for  the  general  user.  The  computer  technologies  for  graphics  and  sound  both  began 
this  way.  Early  in  their  development  the  chipsets  for  professional  quality  results  were  expensive 
and  used  only  in  professional  applications.  As  these  systems  matured  and  processing  power 
available  continued  to  increase,  these  systems  began  to  a  see  a  rise  in  non-specialized 
applications  that  have  led  to  their  rapid  adoption  and  ubiquitous  use  in  every  computer  now  sold. 

VR  should  follow  a  similar  pathway  in  the  evolutionary  gains  toward  a  more  user-friendly 
system.  The  hardware  will  become  less  specialized  and  usable  in  a  variety  of  applications  that 
will  open  the  door  to  greater  flexibility  and  usefulness  by  diverse  sections  of  the  population. 
This  move  to  general  user  interfaces  will  make  the  technology  cheaper  to  field  and  update.  One 
indicator  that  this  may  already  be  happening  is  the  military’s  investment  into  the  GOSE 
simulators  for  maintenance  personnel  training  with  multiple  aircraft  using  the  same  interface.46 
Trend  Analysis 

The  overall  trend  for  VR  interfaces  is  that  they  will  continue  to  develop  better  resolutions 
for  each  channel  and  lighter  and  faster  processing  mechanisms  which  will  provide  enhanced 
immersion  and  interactivity.  The  technology  will  continue  to  advance  along  the  evolutionary 
pathway  toward  improvement  along  individual  channels  of  interface.  There  will  be  some 
convergence  of  technologies  at  the  device  level,  but  overall  the  gains  will  take  place  within  the 
individual  channels  discussed  previously. 

One  of  the  most  significant  developments  for  this  technology  will  be  its  decreased  reliance 
upon  specialized  systems  with  a  move  toward  more  generalized  systems.  The  trend  suggests  that 
future  VR  systems  will  shift  away  from  proprietary  interfaces  for  specific  applications  and 
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toward  modular  software  based  systems  to  interface  with  a  generic  system.  Computer  systems  of 
the  future  may  ship  with  a  basic  VR  interface  as  a  standard  alongside  the  speakers,  mouse,  and 
keyboard.  This  interface  will  likely  be  made  up  of  an  enhanced  tracking  system,  head  mounted 
display  and  haptic  feedback  using  a  vibrotactile  data  glove  for  interactivity.47  Systems  like  this 
will  serve  as  a  bridge  between  current  legacy  computer  systems  and  future  immersive  VR 
configurations.  But  these  evolutionary  gains  will  only  allow  us  to  progress  to  a  certain  point  on 
the  continuum  between  weak  and  strong  VR.  All  of  the  gains  made  available  in  each  channel 
area  will  be  unable  to  overcome  all  of  the  limits  that  are  inherent  in  any  interface  that  must  be 
strapped  onto  a  user’s  body. 

What  does  this  mean  for  the  future  of  virtual  reality  interfaces?  Is  the  “holy  grail”  of  strong 
VR  unattainable?  Beyond  this  evolutionary  pathway  toward  stronger  VR,  are  there  alternate 
pathways  that  may  be  explored  to  provide  revolutionary  disruptive  changes  to  VR  interfaces  by 
2025? 

Alternate  Pathways 

These  limitations  by  the  evolutionary  pathways  toward  better  interfaces  for  each  channel  can 
only  be  overcome  by  trying  to  explore  alternate  pathways  to  the  “holy  grail”  of  VR  interfaces. 
Most  of  the  limitations  of  the  current  solutions  are  based  on  our  limited  knowledge  of  how  our 
sense  organs  translate  their  information  to  the  brain.  A  large  portion  of  the  advances  in  VR 
interface  technology  have  come  by  exploiting  the  limits  of  the  human  senses  -  like  creating 
motion  video  using  30  frames  per  second  -  to  approximate  reality.  As  we  increase  our 
understanding  of  the  inherent  functions  of  the  senses,  we  will  continue  to  increase  our  ability  to 
take  advantage  of  their  limitations.  But  this  still  does  not  solve  the  greatest  limitations  regarding 
strong  VR. 


22 


Using  a  reverse  tree  methodology,  I  have  been  able  to  break  the  possible  solutions  for  VR 
interface  into  two  distinct  pathways.  The  first  solution  is  to  create  external  interfaces  that 
provide  the  proper  stimulation  to  the  user’s  sense  organs  in  order  to  replicate  all  sensations  that 
are  necessary  to  create  the  immersive  and  interactive  environment.  This  solution  set  is 
represented  by  the  evolutionary  pathways  presented  above.  The  second  set  of  solutions  bypasses 
the  external  sense  organs  of  the  body,  and  replicates  the  necessary  messages  on  the  body’s  neural 
pathways  to  fool  the  body  into  believing  that  it  is  experiencing  the  virtual  environment.  These 
solutions  would  be  able  to  overcome  the  majority  of  the  limitations  of  the  first  solution  set,  while 
providing  a  significantly  greater  immersive  and  interactive  experience. 

These  brain-machine  interfaces  have  been  envisioned  in  two  primary  ways:  the  first  is  a 
direct  connection  to  the  human  brain  and/or  neural  system  to  allow  for  direct  input  and  output  of 
the  virtual  reality  information;  the  second  is  an  indirect  connection  to  the  human  brain  through  a 
variety  of  different  mechanisms  such  as  the  electromagnetic  spectrum  and  nanotechnology.  The 
basic  research  required  for  these  operations  is  still  in  its  infancy,  and  its  use  in  devices  for 
development  and  applications  are  primarily  in  the  experimental  phases,  but  there  are  a  number  of 
indicators  that  suggest  that  interfaces  based  on  these  ideas  may  experience  a  rapid  growth  in 
research  and  implementation  over  the  next  15  years. 

Brain-Machine  Interfaces 

If  real  is  what  you  can  feel,  smell,  taste  and  see,  then  'real  ’  is  simply  electrical 

48 

signals  interpreted  by  your  brain  -  Morpheus  (The  Matrix) 

Brain-machine  interfaces  represent  a  revolutionary  leap  in  the  way  that  users  may 

potentially  interface  with  a  computer.  This  technology  is  relatively  new  with  the  first 

demonstration  of  neural  activity  directly  controlling  a  robotic  manipulator  coming  in  1999. 49  But 

it  has  experienced  an  explosive  amount  of  growth  in  the  areas  of  basic  and  applied  research 
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primarily  due  to  its  potential  to  allow  handicapped  users  to  restore  motor  function  to  prosthetic 
limbs. 50  This  impetus  for  research  and  interface  design  should  provide  some  spin-off  technology 
that  will  be  directly  usable  by  the  virtual  reality  community. 

Although  this  type  of  interface  seems  to  be  an  unattainable  leap,  there  are  many  current 
instances  of  the  uses  of  technology  that  will  continue  to  push  our  understanding  of  the  human 
brain  and  its  operations.  Cochlear  implants  have  become  fairly  common  for  people  with 
significant  hearing  loss.  This  technology  is  a  simple  form  of  the  interface  that  has  been 
suggested  with  brain-machine  interfaces.  Hearing  devices  translate  external  sound  information 
and  directly  stimulate  the  auditory  nerve  to  augment  the  natural  ear.51  The  next  step  toward  a 
VR  interface  would  be  to  replace  the  amplified  real  sounds  with  computer  generated  sounds. 

There  are  other  examples  of  how  research  is  being  conducted  to  help  us  understand  the  brain 
and  neural  systems  with  the  intent  to  control  via  computer  or  other  device  that  have  direct 
applications  in  VR  interfaces.  Some  individuals  have  been  able  to  implant  a  chip  in  the  neurons 
on  their  arm  to  record  the  neural  messages  generated  by  the  brain  to  command  hand  movements. 
They  were  then  able  to  play  back  these  recorded  impulses  and  have  the  computer  command  the 
same  movements  exactly  as  recorded.' “  Other  researchers  have  implanted  controller  chips  in  the 
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brains  of  cockroaches  to  allow  them  to  “control”  the  movement  of  the  cockroach  in  a  maze. 
Another  set  of  research  has  focused  on  streaming  video  directly  into  the  visual  cortex  of  blind 
persons  with  limited  success  in  repairing  blindness.54  One  company  is  already  testing  the 
BrainGate™  system  to  enable  users  to  control  a  computer  and  its  attached  devices  through  a  chip 
implanted  on  the  motor  cortex  of  the  brain.55 

In  addition  to  these  experiments  with  invasive  connections  to  the  brain,  other  research  has 
concentrated  on  non-invasive  techniques  to  transfer  brain  functions.  This  research  has  centered 
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on  the  use  of  EEG  and  fMRI  technology  to  sample  the  activity  of  the  human  brain  through 
sensors  outside  the  body  for  control  of  computer  cursors  and  mechanical  manipulators.56  Ray 
Kurzweil,  a  noted  technology  futurologist,  forecasts  that  non-invasive  “holy  grail”  VR  interface 
will  be  available  utilizing  nanobot  based  implants  that  will  serve  as  the  direct  connection  to  the 
brain.  The  input  and  output  will  be  conducted  via  communicating  with  the  nanobots  wirelessly 
through  the  skull.57 

Although  some  authors  doubt  that  our  knowledge  of  brain  operations  and  the  ability  for  us  to 
connect  to  and  decipher  neural  pathways  will  ever  be  good  enough  to  support  these  interfaces, 
there  are  a  number  of  experts  in  the  field  that  have  forecast  that  interfaces  with  the  brain  could  be 
available  as  early  as  2020.  A  RAND  study  of  future  technology  labeled  chip  implants  in  the 
human  brain  as  one  of  its  “wild-card”  technologies  for  2020.  Most  of  the  basic  research  needed 
to  implement  applied  research  and  development  could  be  available  as  early  as  20 10. 59  These 
timelines  should  only  be  helped  by  the  increased  emphasis  on  current  research  into  these  areas, 
as  well  as  the  exponential  growth  of  computer  processing  power  to  help  solve  the  current  issues. 
This  technology  should  be  available  in  the  future60  and  would  provide  an  extremely  valuable 
advantage  to  anyone  who  is  able  to  utilize  it. 

Analysis  and  Recommendation 

Using  the  technological  trends  forecast  for  the  current  VR  interface  technology  and  the 
potential  promise  of  brain-machine  interfaces,  I  have  created  future  vectors  for  each  of  the 
technologies  using  the  VR  framework  diagram  (Figure  5).  There  is  no  doubt  that  given  the 
increasing  perfonnance  of  computing  technologies  in  general  and  the  increasing  attention  being 
paid  to  VR  interfaces,  both  in  research  dollars  and  systems  fielded,  should  lead  this  evolutionary 
pathway  closer  to  the  concept  of  strong  VR.  But  the  limitations  inherent  in  the  current  iteration 
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of  VR  interface  technology  will  limit  the  ability  of  these  interfaces  to  achieve  the  “holy  grail”  of 
immersive  and  interactive  environments  indistinguishable  from  everyday  reality.  This  does  not, 
however,  imply  that  the  VR  interfaces  in  use  today  will  not  have  a  place  in  our  operations  in 
2025. 


Interactive 


Figure  5  Location  of  VR  interface  devices  and  prospects  for  the  future  on  Immersive/Interactive  Continuum 
On  the  contrary,  they  will  most  likely  be  the  standard  method  of  interface  with  most  desktop 
computing  systems  for  knowledge  retrieval  and  manipulation.  These  current  technologies  will 
continue  to  become  faster,  cheaper,  and  more  user-friendly  with  each  new  generation.  They  will 
make  the  leap  from  specialized  hardware  for  specific  applications  to  generalized,  modular 
software-based  systems  that  will  create  new  opportunities  for  applications  in  all  areas  of 
computer  training  and  data  visualization.61  These  generalized  interfaces  will  not  gain  market 
saturation  until  a  “killer  application”  is  created  that  drives  the  need  for  the  technology  to 
proliferate.  The  emergence  of  this  “killer  application”  will  signal  the  maturation  of  these  VR 
interface  devices  for  use  in  standardized  systems. 
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The  size  of  the  VR  interface  industry  and  the  market  for  many  different  types  of  VR  modes 
and  applications  will  continue  to  allow  all  of  the  current  interfaces  to  occupy  their  own  niche  of 
the  market.  This  will  spur  enhancements  for  each  of  the  current  technologies  and  even  provide 
overlap  for  competing  technologies  within  the  market  space.  No  one  interface  will  be  able  to 
dominate  the  entire  market  based  on  the  requirements  for  specialized  interfaces  in  each 
application.  In  addition,  the  interfaces  will  continue  to  be  dispersed  at  all  levels  of  spending  in 
the  market,  from  cheap  home-use  VR  interfaces,  to  expensive  application  specific  hardware  and 
software  interfaces.  But  in  order  to  achieve  “holy  grail”  strong  VR  interfaces,  a  new 
revolutionary  pathway  will  have  to  emerge. 

This  alternate  pathway  for  future  development  will  have  to  be  able  to  overcome  the  physical 
limitations  of  the  current  VR  systems,  as  well  as  the  future  systems  they  will  become.  The  most 
likely  candidates  for  this  revolution  in  technology  are  found  in  the  area  of  brain-machine 
interfaces.  These  systems  allow  the  technology  to  overcome  all  of  the  physical  barriers  by 
bypassing  the  human  sensory  organs.  This  pathway  contains  both  invasive  and  non-invasive 
methods  for  achieving  direct  connection/stimulation  of  the  human  neural  system.  Although  the 
challenges  in  researching  and  fielding  this  technology  are  many  and  varied,  “  the  basic  and 
applied  research  continues  to  grow  at  an  increasing  rate. 

A  strong  VR  interface  would  provide  a  significant  military  advantage  for  its  user  in  a 
number  of  disparate  areas.  Training  of  any  skills  would  become  cheaper  and  more  effective,  but 
the  US  Air  Force  would  most  likely  benefit  the  most  from  gains  in  flying  training.  As  fuel 
becomes  more  expensive  and  training  ranges  more  restrictive,  strong  VR  would  allow  us  to  shift 
the  bulk  of  our  currency  training  to  the  virtual  realm  with  little  drop  in  proficiency.  Mission 
rehearsals  could  be  carried  out  in  the  exact  conditions  expected  for  any  possible  mission.  This 
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increases  the  likelihood  of  success  and  could  expose  advantages  previously  unknown  against 
enemy  systems.  Virtual  reality  data  visualization  will  allow  a  more  natural  situational  awareness 
tool  that  will  allow  our  decision  cycles  to  be  shortened  within  an  increasingly  information¬ 
intensive  battlefield.  Finally,  other  spin-off  technologies,  like  chip  implants  in  the  brain,  may 
create  asymmetric  advantages  at  the  individual  soldier  level  that  will  enhance  the  entire  force. 

The  US  Air  Force  should  continue  to  invest  in  research  and  development  of  the  current  VR 
interface  technologies  for  specific  applications.  We  should  use  off-the-shelf  developments  from 
the  entertainment  industry  to  the  maximum  extent  to  rapidly  enhance  our  military  systems.  This 
will  ensure  that  we  maintain  an  advantage  in  training  and  simulation  for  as  long  as  possible, 
while  leveraging  the  entertainment  industry  advances  to  control  short  term  costs.  We  should 
develop  a  standard  desktop  configuration  VR  interface  that  enables  software  to  be  developed  to 
enhance  our  current  training  programs  with  modular,  software-based  applications.  In  addition, 
we  must  actively  seek  out  the  “killer  application”  for  military  uses  of  this  technology.  This 
application  may  be  tied  to  our  ever  increasing  information  overload  and  this  technology  could 
provide  us  an  interface  that  is  more  user-friendly  under  these  data-intensive  operations. 

In  addition  to  financing  developments  in  the  evolutionary  pathways,  the  US  Air  Force 
should  invest  in  basic  research  of  the  human  brain  and  neural  systems  with  a  goal  of  mapping 
brain  functions  and  neural  pathways.  We  should  fund  applied  research  and  development  for 
brain-machine  interfaces  that  enhance  our  ability  to  provide  stronger  VR  interfaces.  We  must 
encourage  the  VR  interface  community  to  continually  push  to  increase  its  ability  to  deliver 
highly  immersive  and  infinitely  interactive  virtual  environments  that  can  accurately  portray 
common  Air  Force  mission  and  tasks. 
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Virtual  reality  is  already  in  widespread  use  in  basic  applications  throughout  the  US  Air 
Force.  These  VR  systems  depend  upon  the  interfaces  to  provide  users  with  an  immersive, 
interactive  environment  that  closely  approximates  reality.  The  future  trend  is  that  VR  will 
become  more  useful  and  more  accessible.  These  advances  will  be  enabled  by  the  growth  of 
computer  processing  power,  the  money  spent  on  research  and  development  of  these  systems,  and 
better  understanding  of  the  human  sensory  systems.  But  these  evolutionary  advances  will  not  be 
able  to  achieve  “holy  grail”  strong  VR.  Brain-machine  interfaces  represent  a  feasible 
technological  pathway  to  achieve  an  immersive,  interactive  computer-generated  environment 
that  is  indistinguishable  from  reality.  This  level  of  VR  would  create  a  significant  military 
advantage  for  the  US  Air  Force  that  would  enable  them  to  maintain  superiority  in  training  and 
simulations  for  the  foreseeable  future. 
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Appendix  A:  Related  Technologies 


Although  the  focus  of  this  paper  was  almost  solely  focused  on  virtual  reality  and  the  human- 
machine  interface  required  for  it,  there  are  many  closely  related  technologies  that  would  also 
benefit  from  an  improved  human-machine  interface.  These  technologies  each  have  their  own 
areas  that  they  are  best  suited  for,  and  they  provide  increased  impetus  for  us  to  pursue  an 
interface  that  is  closer  to  strong  VR.  Most  of  them  have  a  limited  use  in  the  US  Air  Force 
currently,  but  their  use  would  be  greatly  expanded  with  improvements  to  interface  technology. 

Augmented  reality  (or  mixed  reality)  uses  interface  hardware  similar  to  virtual  reality  to 
overlay  computer  generated  images  onto  reality  in  real  time.  It  adds  digital  information  to  the 
everyday  scene. 1  A  HUD  is  an  example  of  an  augmented  reality  that  can  be  used  to  help  pilots 
understand  more  about  their  environment  around  them  through  the  use  of  computer  information. 
With  an  enhanced  user  interface,  this  technology  could  be  expanded  to  be  used  to  overlay 
maintenance  diagrams  directly  over  the  aircraft  system  to  provide  better  maintenance 
performance."  It  may  also  be  used  to  enhance  the  situational  awareness  of  airspace  controllers 
(in  a  control  tower)  by  overlaying  aircraft  infonnation  as  they  scan  their  airspace. 

Teleoperation  uses  similar  human-computer  interfaces  to  allow  the  user  to  remotely  operate 

•> 

a  piece  of  equipment  as  if  they  were  physically  manipulating  it.  It  may  be  considered  an 
offshoot  of  virtual  reality  that  provides  the  same  interface  with  the  computer,  but  the 
environment  is  not  created  by  the  computer,  it  is  merely  communicated  through  the  computer. 
The  primary  difference  between  this  systems  and  a  VR  system  is  the  teleoperation  system  must 
have  the  capacity  to  accurately  communicate  conditions  as  they  actually  are,  which  may  require 
a  significant  amount  of  bandwidth.  These  types  of  systems  are  in  use  today  by  surgeons  who  are 
able  to  operate  on  patients  remotely  as  if  they  were  in  the  same  location. 


34 


Teletravel  and  telepresence  are  also  related  technologies  that  allow  a  user  to  interact  with 
other  users  in  a  different  location  as  if  they  were  together.  It  is  the  next  generation  in  “virtual 
teleconference”  applications.  These  technologies  rely  on  the  full  sensory  experience  to  allow 
individuals  to  hold  meetings  while  physically  separated  that  approximate  the  experience  of  being 
in  the  same  room.4 

In  addition  to  these  related  technologies  that  are  in  development  today,  VR  interface 
research  could  create  new  spin-off  technologies.  One  possible  area  for  spin-offs  in  the  brain- 
machine  interface  is  in  the  area  of  chip  implants  in  the  human  brain.5  Chip  implants  would  allow 
human  soldiers  to  enhance  their  ability  to  store  and  process  information.  They  may  also  allow 
for  increasing  the  physical  abilities  of  the  user  by  stimulating  the  proper  areas  of  the  brain  to 
achieve  the  desired  effect. 
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Appendix  B:  Military  Applications 


The  military  already  uses  virtual  reality  extensively  in  its  training  and  simulation.  These 
efforts  began  with  flight  simulators  and  have  recently  been  expanded  to  include  simulators  for 
other  disciplines  like  convoy  driving  and  cultural  training.6  These  applications  allow  the  military 
to  put  their  personnel  into  realistic  situations  that  can  sharpen  their  skills  and  help  them  perform 
at  their  peak  during  the  real  task.  The  bulk  of  the  simulators  in  use  for  training  use  specialized 
systems  that  utilize  a  real  object  to  act  as  the  interface  between  the  user  and  the  virtual  world. 
This  object  provides  the  physical  feedback  and  link  between  the  training  and  the  objects  use  in 
the  real  world.7 

An  extension  of  using  virtual  reality  for  training  is  to  use  it  as  a  mission  rehearsal  device. 
The  application  of  virtual  reality  would  increase  the  chances  of  success  for  any  mission  by 
allowing  the  users  to  rehearse  their  mission  in  the  exact  conditions  that  would  be  found  during 
the  actual  mission.  In  addition,  the  virtual  world  would  allow  them  to  simulate  a  variety  of 
changing  factors  to  help  the  planners  be  prepared  for  any  other  situations  that  could  wreak  havoc 
with  their  plans.  Virtual  mission  rehearsal  would  also  increase  security  by  not  alerting  the 
enemy  to  the  United  States’  preparation  for  missions  in  a  particular  location.  A  “virtual  Sontay” 
would  have  allowed  the  raiding  force  to  practice  their  assault  on  the  prison  camp  while 
simulating  various  changes  to  the  environment.  The  “virtual  Sontay”  would  have  been  secure 
from  the  prying  eyes  of  the  enemy  and  the  element  of  surprise  would  have  been  maintained 
without  having  to  take  down  the  structures  every  time  Soviet  satellites  were  overhead.  Using 
this  same  logic,  the  United  States  could  protect  its  tactics,  techniques,  and  procedures,  as  well  as 
its  strategy,  while  continually  exercising  them  to  learn  their  strengths  and  weaknesses. 
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Augmented  reality  is  another  offshoot  application  of  virtual  reality  interfaces  that  promises 
tremendous  gains  for  the  military.  Heads-up  displays  (HUD)  are  a  rudimentary  augmented 
reality  display  that  provides  information  about  the  battlespace  as  an  overlay.  More  recent 
cockpit  upgrades  have  begun  to  create  a  HUD  that  is  not  mounted  on  the  aircraft,  but  is 
embedded  in  the  visor  of  the  pilot.  This  technology  may  also  be  expanded  for  use  by 
maintenance  personnel  on  the  ground.  Augmented  reality  would  allow  the  maintainer  to  overlay 
part  and  supply  information  while  looking  at  an  aircraft  system.  It  would  also  allow  them  to 
have  repair  instructions  overlaid  on  the  actual  job  as  they  perform  it.  This  application  would 
allow  more  efficient  use  of  our  maintenance  personnel  while  simultaneously  ensuring  the  job 

was  being  performed  accurately.  An  augmented  reality  system  is  already  in  development  for 

8 

training  C-130  loadmasters  at  Little  Rock. 

One  of  the  most  promising  applications  for  virtual  reality  is  in  data  visualization  and  data 
analysis.  Large  amounts  of  data  are  being  collected  by  the  increasing  number  and  type  of  sensor 
platforms  on  the  battlefield.  This  explosion  of  information  that  is  being  collected  will  continue 
to  increase  as  sensors  become  smaller,  cheaper,  and  more  capable.  The  amount  of  information  is 
rapidly  outpacing  the  human  ability  to  sort  it  and  make  sense  of  it.  Virtual  reality  can  provide  a 
more  natural  interface  for  the  user  to  interact  with  complex  data  -  allowing  them  to  delve  deeper 
into  the  data  to  extract  meaningful  information.9  Some  of  the  more  useful  applications  for  this 
technology  would  be  to  allow  a  “Virtual  Common  Operating  Picture”  for  the  JFACC  and  the 
AOC  to  analyze  the  battlespace  in  3D.  Airspace  controllers  could  also  use  this  technology  to 
fuse  all  of  their  information  into  “virtual  airspace”  to  help  them  make  the  best  decision  with  all 
of  the  information  in  a  more  intuitive  display.  These  two  applications  would  allow  the  users  to 
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“see”  information  like  closure  rates,  proximity  to  restricted  airspace,  instrument  approaches,  and 
weapons  flight  paths  and  to  analyze  their  effects  to  operations. 

Another  application  of  VR  interface  technology  that  could  enhance  military  acquisitions  is 
virtual  prototyping.10  This  application  allows  a  designer  or  engineer  to  model  a  new  weapon 
system  or  control  mechanism  in  virtual  space.  The  users  could  then  operate  the  new  system  in  a 
realistic  environment  to  provide  feedback  on  the  usability  of  the  system.  Virtual  prototyping 
allows  most  design  changes  in  a  system  to  be  made  prior  to  fabrication  of  any  pieces  of  the 
system.  This  decreases  the  time  needed  for  engineering  and  development,  while  simultaneously 
ensuring  the  finished  product  is  more  suitable  in  the  first  iteration. 
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Appendix  C:  Non-Technical  Factors 

Although  the  technical  hurdles  for  a  working  brain-machine  interface  are  considerable,  they 
may  not  be  the  biggest  hurdles  overcome.  There  are  other  non-technical  factors  that  may  affect 
the  development  of  brain-machine  interfaces  in  the  future. 

Social 

Brain-machine  interfaces  may  be  difficult  to  implement  unless  people  are  willing  to  accept  a 
technology  that  is  intimately  tied  to  the  functions  of  their  brain.  Although  we  are  generally 
accepting  of  computer  technology  within  our  lives,  most  people  realize  that  computer  glitches 
are  not  uncommon.  Some  users  may  be  reluctant  to  use  any  technology  that  interferes  with  their 
neural  control  of  their  body.  Fail-safe  mechanisms  and  less  error-ridden  code  must  be  developed 
to  overcome  societal  resistance  to  brain-machine  interfaces. 

Brain  surgery  stands  as  another  huge  barrier  for  direct-connection  brain-machine  interface 
technology.  The  surgery  required  to  implement  an  invasive  brain-machine  interface  would  be  a 
risky  surgery  that  would  entail  a  tremendous  period  of  recovery  and  training  to  enable  error- free 
use.  The  cost  in  money  and  hospital  procedures  without  significant  gains  from  the  surgery 
would  deter  the  majority  of  users  from  having  it  done.  But  some  experts  believe  that  this  societal 
reluctance  may  be  overcome  easier  in  future  generations.  They  point  out  that  there  is  already  a 
tremendous  industry  of  surgical  self-modification,  especially  when  the  surgery  can  provide  a 
professional  advantage  (memory  or  attention-span).11 

Although  one  could  argue  either  way  about  individual  choice,  the  individuals  may  never 
have  the  ability  to  make  that  choice.  A  RAND  study  stated  that  public  policy  issues  will  be  the 
biggest  damper  to  their  development.  The  barriers  to  acceptance  by  government  regulators  may 
be  the  biggest  hurdle  for  these  interfaces  to  overcome.  '  It  is  difficult  enough  for  new  drugs  or 
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plastic  surgery  procedures  to  be  approved  within  our  regulations,  but  an  elective  surgery  to 
implant  a  chip  or  connection  within  a  human  brain  may  take  decades  to  convince  the  government 
that  its  use  and  implementation  does  not  represent  a  health  risk  to  the  public. 

Non-invasive  brain-machine  interface  solutions  will  be  the  easiest  to  assimilate  into  our 
culture.  Our  culture  is  already  willing  to  hang  technology  from  our  bodies  if  it  provides  us  an 
advantage  and  we  can  remove  it  when  not  needed  (cell  phones  and  watches).  Solutions  for 
brain-machine  interfaces  that  do  not  require  a  permanent  modification  of  the  human  physical 
condition  may  be  used  much  like  other  technology  that  we  accept  on  a  daily  basis.  But  this 
assumes  either  no  need  for  nanobots  or  other  foreign  material  in  the  human  neural  system  - 
otherwise  the  barriers  are  the  same  for  invasive  connections. 

Moral /Ethical 

In  addition  to  the  medical  and  physical  barriers  that  VR  connections  may  encounter,  the 
ability  to  create  strong  VR,  indistinguishable  from  reality,  may  expose  moral  and  ethical  issues 
that  our  culture  or  legal  system  is  not  equipped  to  handle.  These  issues  may  blur  our 
understanding  of  morality  by  creating  virtual  infidelity  and  virtual  adultery.  Our  culture  may 
have  to  come  to  an  understanding  of  what  is  morally  acceptable  behavior  in  virtual  environments 
that  are  perfect  substitutes  for  reality.  We  may  also  be  faced  with  ethical  issues  surrounding 
aberrant  behaviors  carried  out  in  virtual  environments.  Does  the  murder  of  virtual  characters 
increase  the  likelihood  of  the  user  carrying  out  violent  acts  in  reality?  These  issues  are  not 
sharply  defined,  nor  can  they  be  forecast  enough  to  provide  legislation  to  answer  these  questions 
for  our  entire  country.  The  ethical  debate  surrounding  violent  video  games  and  their  effect  on 
young  people  is  one  indicator  that  these  issues  are  not  easily  understood.  The  outcome  of  these 
problems  may  present  a  large  hurdle  for  widespread  acceptance  of  strong  VR  interfaces. 
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Psychological  /  Physiological 

Current  VR  interfaces  are  limited  in  their  use  due  to  physiological  limitations  of  the  human 
body.  Some  users  of  current  technology  have  experienced  VR  motion  sickness,  caused  when  the 
visual  cues  in  the  virtual  environment  do  not  match  with  the  vestibular  cues  of  the  user.  The 
brain  is  unable  to  decide  which  of  the  systems  is  not  correct  and  the  result  is  physical  illness  and 
may  be  accompanied  with  vomiting.  This  sickness  is  a  result  of  the  current  interfaces’  inability 
to  overcome  its  physical  limitations  as  discussed  in  the  channels  section.  Strong  VR  (the  “holy 
grail”)  should  not  have  this  same  problem  because,  in  theory,  all  of  the  user’s  senses  would 
experience  the  proper  cues  for  the  virtual  environment  and  the  body  would  not  experience  any 
disconnects  between  senses. 

This  sub-area  also  is  closely  related  to  the  social  discussion  above.  There  would  be  a 
lengthy  process  for  research  and  development  of  any  invasive  VR  technology  or  non-invasive 
technology  that  utilized  nanobot  technology  for  communication.  The  US  government  would 
serve  as  a  major  barrier  for  this  technology  due  to  its  regulatory  oversight  for  any  medical 
procedures  of  this  nature.  But  this  may  only  serve  as  a  barrier  in  the  US  and  nations  with  similar 
policies  regarding  medical  testing.  If  a  nation  believes  that  the  gains  from  research  of  human- 
machine  interfaces  would  be  significant,  it  may  have  an  incentive  to  override  its  normal 
procedures.  Furthermore,  not  all  nations  have  the  same  restrictions  on  testing  and  research.  This 
may  create  an  opportunity  for  an  emerging  nation  to  gain  critical  technological  insight  ahead  of 
the  rest  of  the  international  community. 
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